Adaptive setting of the quantized congestion notification equilibrium setpoint in converged enhanced eithernet networks

ABSTRACT

Embodiments relate to controlling workload flow on converged Ethernet links. An aspect includes coupling, by a processing device, a first control loop to a second control loop. The second control loop monitors the operation of the first control loop. An equilibrium set point is initialized for the second control loop prior to commencing operation of the first control loop. Accordingly, the equilibrium set point value is adjusted in the second control loop continuously based on a rate of operation of the first control loop.

DOMESTIC PRIORITY

This application is a continuation of U.S. patent application Ser. No.13/834,600, filed Mar. 15, 2013, the content of which is incorporated byreference herein in its entirety.

BACKGROUND

The present invention relates to network congestion, and morespecifically, to adaptively determining the equilibrium set point for aquantized congestion notification (QCN) protocol to prevent instabilityor a race condition between a priority flow control (PFC) protocol and aQCN protocol on converged Ethernet links. The present invention may alsoapply to virtualized CEE-like Software Defined Networks (SDN).

IEEE 802.1 data center bridging (DCB) Ethernet is often used as aprimary physical network protocol in datacenters forcomputer-to-computer communications. In contrast to the modern 802.1QauDCB (i.e., convergence enhanced Ethernet (CEE)/data center Ethernet(DCE)), the traditional Ethernet was typically designed to be abest-effort network and may experience network congestion. Networkcongestion occurs when a link or node is carrying so much data that itsquality of service deteriorates. Typical effects include queueing delay,packet loss and/or the blocking of new connections. A consequence ofpacket loss or the blocking of new connections is that incrementalincreases in offered load only lead to small increases in networkthroughput or to an actual reduction in network throughput. Moreover,network congestion may lead to a congestive collapse at a choke point inthe network, where the total incoming traffic to a node exceeds theoutgoing bandwidth. When a network is in such a condition, there arehigh levels of packet delay and loss and general quality of networkservice is poor. These effects are aggravated in the modern multitenantdatacenters using server and network virtualization on top of Layer 2CEE fabrics with Priority Flow Control (PFC, 802.1Qbb), used forconverged lossless traffic. Hence the need for Quantized CongestionNotification (QCN, 802.1Qau), which is today built in most modern CEEswitches and adapters, as well as being introduced in the new VirtualOverlays and software-defined networking (SDN) solutions.

SUMMARY

According to an embodiment of the present invention, a method forcontrolling workload flow on converged Ethernet links is provided. Themethod includes coupling, by a processing device, a first control loopto a second control loop. The second control loop monitors the operationof the first control loop. An equilibrium set point is initialized forthe second control loop prior to commencing operation of the firstcontrol loop. Accordingly, the equilibrium set point value is adjustedin the second control loop continuously based on a rate of operation ofthe first control loop.

According to another embodiment of the present invention, a computerprogram product for controlling workload flow on converged Ethernetlinks is provided. The computer program product includes a storagemedium having computer-readable program code embodied thereon, whichwhen executed by a computer processor, causes the computer processor toimplement a method. The method includes coupling a first control loop toa second control loop. The second control loop monitors the operation ofthe first control loop. An equilibrium set point is initialized for thesecond control loop prior to commencing operation of the first controlloop. Accordingly, the equilibrium set point value is adjusted in thesecond control loop continuously based on a rate of operation of thefirst control loop.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention. For a better understanding of the invention with theadvantages and the features, refer to the description and to thedrawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The forgoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 depicts a cloud computing node according to an embodiment;

FIG. 2 depicts a cloud computing environment according to an embodiment;

FIG. 3 depicts abstraction model layers according to an embodiment; and

FIG. 4 depicts a flow diagram of an adaptive equalization process fordynamically determining an equilibrium set point for a quantizedcongestion notification protocol according to an embodiment.

DETAILED DESCRIPTION

Embodiments disclosed herein allow a workload flow to increase at acontrolled rate while preventing instability on converged Ethernetlinks, the instability being caused by a race condition between apriority flow control (PFC) protocol and a quantized congestionnotification (QCN) protocol. Embodiments provide an adaptiveequalization operation for dynamically determining an equilibrium setpoint for a QCN protocol. Embodiments couple the PFC protocol to the QCNprotocol so that the QCN protocol is able to monitor the operation ofthe PFC protocol and adjust the equilibrium set point value to ensurethat the QCN protocol will always activate prior to the PFC protocol.

In a contemporary Ethernet fabric, the PFC protocol and the QCN protocoloperate independently from one another. Accordingly, the PFC and QCNprotocols may compete with one another and get stuck in a racecondition. As the network gets congested, sometimes QCN kicks in firstand sometimes PFC kicks in first. Further, when both protocols are neartheir threshold, one protocol may kick in and runs for a while, thenshut off to allow the other protocol to kick in. This causes a racecondition of which protocol is going to take over first and createsinstability in the network due to link flapping. This is especially aproblem if the data patterns going through the network are random andunpredictable. Network performance may suffer and, in the worst case, acatastrophic collapse of the network may occur due to unsteady packetflow, which shuts the link down.

Accordingly, embodiments couple the PFC and QCN protocols to provide anadaptive equalization operation for dynamically determining anequilibrium set point for a QCN protocol and ensure that the QCN(stable, yet slower e2e) protocol will always have the chance toactivate prior to the PFC (faster, yet prone to oscillations link-level)protocol.

It is understood in advance that although this invention includes adetailed description on cloud computing, implementation of the teachingsrecited herein are not limited to a cloud computing environment. Rather,embodiments of the invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes.

Referring now to FIG. 1, a block diagram of a cloud computing node foradaptively determining the equilibrium set point for a QCN protocol ofan embodiment is shown. Cloud computing node 10 is only one example of asuitable cloud computing node and is not intended to suggest anylimitation as to the scope of use or functionality of embodimentsdescribed herein. Regardless, cloud computing node 10 is capable ofbeing implemented and/or performing any of the functionality set forthhereinabove.

In cloud computing node 10 there is a computer system/server 12, whichis operational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 12 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 12 may be described in the general context ofcomputer system-executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 12 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 1, computer system/server 12 in cloud computing node 10is shown in the form of a general-purpose computing device, alsoreferred to as a processing device. The components of computersystem/server 12 may include, but are not limited to, one or moreprocessors or processing units 16, a system memory 28, and a bus 18 thatcouples various system components including system memory 28 toprocessor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus.

Computer system/server 12 may include a variety of computer systemreadable media. Such media may be any available media that is accessibleby computer system/server 12, and it includes both volatile andnon-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in memory 28 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments as described herein.

Computer system/server 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computersystem/server 12; and/or any devices (e.g., network card, modem, etc.)that enable computer system/server 12 to communicate with one or moreother computing devices. Such communication can occur via Input/Output(I/O) interfaces 22. Still yet, computer system/server 12 cancommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20. As depicted, network adapter 20communicates with the other components of computer system/server 12 viabus 18. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 12. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

Referring now to FIG. 2, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 comprises one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,digital video camera 54D, digital audio recording device 54E, and/ordigital still camera 54N may communicate. Nodes 10 may communicate withone another. They may be grouped (not shown) physically or virtually, inone or more networks, such as Private, Community, Public, or Hybridclouds as described hereinabove, or a combination thereof. This allowscloud computing environment 50 to offer infrastructure, platforms and/orsoftware as services for which a cloud consumer does not need tomaintain resources on a local computing device. It is understood thatthe types of computing devices 54A-N shown in FIG. 2 are intended to beillustrative only and that computing nodes 10 and cloud computingenvironment 50 can communicate with any type of computerized device overany type of network and/or network addressable connection (e.g., using aweb browser).

Referring now to FIG. 3, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 2) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 3 are intended to be illustrative only and embodiments are notlimited thereto. As depicted, the following layers and correspondingfunctions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include mainframes, in oneexample IBM® zSeries® systems; RISC (Reduced Instruction Set Computer)architecture based servers, in one example IBM pSeries® systems; IBMxSeries® systems; IBM BladeCenter® systems; storage devices; networksand networking components. Examples of software components includenetwork application server software, in one example IBM WebSphere®application server software; and database software, in one example IBMDB2® database software. (IBM, zSeries, pSeries, xSeries, BladeCenter,WebSphere, and DB2 are trademarks of International Business MachinesCorporation registered in many jurisdictions worldwide).

Virtualization layer 62 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers;virtual storage; virtual networks, including virtual private networks;virtual applications and operating systems; and virtual clients.

In one example, management layer 64 may provide the functions describedbelow. Resource provisioning provides dynamic procurement of computingresources and other resources that are utilized to perform tasks withinthe cloud computing environment. Metering and Pricing provide costtracking as resources are utilized within the cloud computingenvironment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal provides access to the cloud computing environment forconsumers and system administrators. Service level management providescloud computing resource allocation and management such that requiredservice levels are met. Service Level Agreement (SLA) planning andfulfillment provides pre-arrangement for, and procurement of, cloudcomputing resources for which a future requirement is anticipated inaccordance with an SLA.

Workloads layer 66 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation; software development and lifecycle management; virtualclassroom education delivery; data analytics processing; transactionprocessing; and network control processing 67.

Turning now to FIG. 4, an adaptive equalization process 400 for tuningthe QCN control loop based on tracking the PFC loop is performed by thenetwork congestion processing module 67 is generally shown. As shown inFIG. 4, the adaptive equalization process 400 dynamically determines anequilibrium set point for a QCN protocol. An embodiment adaptivelydetermines the equilibrium set point for a QCN protocol to maximize thelink utilization, increase stability and prevent a race conditionbetween a PFC protocol and a QCN protocol on converged Ethernet links.

At block 410, an embodiment couples the PFC protocol to the QCNprotocol. By coupling the PFC protocol to the QCN protocol, the QCNprotocol is able to monitor the operation of the PFC protocol. Anembodiment may be implemented using coupled control feedback loops. Forexample, a first control loop (PFC) may be coupled to a second controlloop (QCN) whereby the second control loop monitors the operation of thefirst control loop.

At block 420, an embodiment sets a low equilibrium set point (Qeq) valuefor the QCN protocol. The Qeq is the target level around which a queuelength should oscillate under normal congestion conditions. According toan embodiment, the Qeq threshold is set at a lower value thancontemporary systems and QCN standard Qeq recommendations of ca. 20% ofQmax to ensure that the QCN protocol will always be activated prior tothe PFC protocol. For example, the Qeq may be initially set to a lowvalue to trigger the QCN to transmit a congestion notification messagewhen a queue reaches a range anywhere from approximately 0.5-50%capacity. In contrast, contemporary systems set the Qeq value staticallyat 17-25% based on the size of the queue Qmax. Accordingly, incontemporary systems, the QCN will generally wait until the queuefeedback exceeds a fixed value calculated based on the static Qeq beforesending a congestion notification message (CNM).

By dynamically lowering the Qeq value, an embodiment improves theoverall utilization and prevents a race condition between the PFCprotocol and the QCN protocol on converged Ethernet links by triggeringthe QCN protocol before the PFC protocol. According to an embodiment,the Qeq is initialized for the second control loop (QCN) prior tobeginning operation of the first control loop (PFC). Because the queuesof the switches in the network can absorb more packets before needing tothrottle a source data rate, it would be inefficient to leave the Qeq atthe low value shown in block 420.

Accordingly, at block 430, the second control loop (QCN) monitors theoperation of the first control loop (PFC). Based on the monitoring ofthe first control loop (PFC), an embodiment dynamically adjusts the Qeqin the second control loop (QCN) continuously based on the rate at whichthe first control loop (PFC) operates thereby allowing a workload flowto increase at a controlled rate, as shown in block 440. The dynamicadjustment of the Qeq value may be implemented by a third control loopaccording to an embodiment. The low value of Qeq, which was initiallyset in block 420, is slowly increased based on the monitoring of thefirst control loop (PFC) according to a Qeq control algorithm. The Qeqcontrol algorithm of an embodiment slowly increases the Qeq value untilthe point that the PFC protocol is triggered, thus causing On/Offoscillations typical of such link-level flow control loops.

In block 450, when a flag indicating that the PFC protocol is about tothe triggered is encountered, the control algorithm stops increasing theQeq value, as shown in block 460, and returns to monitoring the activityof the first loop (PFC), as shown in block 430. In other words, anembodiment dynamically determines the point where PFC protocol istriggered and runs the QCN protocol a little below that so that the QCNprotocol will always have the chance to be triggered prior to the PFCprotocol.

If, however, the flag indicating that the PFC protocol is about to thetriggered is not encountered, the control algorithm will continue todynamically adjust the Qeq in the second control loop (QCN) based on therate at which the first control loop (PFC) operates thereby allowing aworkload flow to increase at a controlled rate, as shown in block 440.

According to an embodiment, the Qeq is continuously adapted to be a slowfollower (with a fixed or variable lag) based on the PFC protocolactivity. For example, if a couple of extra switches are added to thenetwork and/or the PFC triggering values changes, the Qeq controlalgorithm of an embodiment automatically increases the Qeq value tocompensate for the PFC value changes due to the added switches in thenetwork. Each control loop of an embodiment has certain parametersassociated therewith, which may be utilized and manipulated by one ofordinary skill in the art to optimize network efficiency, such as thedelay time in the feedback loop, the threshold level, the stabilityconditions, and the like

The control algorithm of an embodiment may be run locally on each switchof the network as well as remotely on a central controller. According toan embodiment, the control algorithm may be implemented usingsoftware-defined networking (SDN) approach (e.g., OpenFlow) thatseparates the central controller that makes decisions about wheretraffic is sent, the control plane, from the underlying switches thatforward traffic to the selected destination, the data plane. Thisarchitecture allows network administrators to have programmable centralcontrol of network traffic without requiring physical access to thenetwork's hardware devices.

Technical effects and benefits of embodiments include allowing aworkload flow to increase at a controlled rate while preventinginstability on converged Ethernet links caused by a race conditionbetween a PFC protocol and a QCN protocol.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present disclosure has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the disclosure in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the disclosure. Theembodiments were chosen and described in order to best explain theprinciples of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosurefor various embodiments with various modifications as are suited to theparticular use contemplated.

Further, as will be appreciated by one skilled in the art, aspects ofthe present disclosure may be embodied as a system, method, or computerprogram product. Accordingly, aspects of the present disclosure may takethe form of an entirely hardware embodiment, an entirely softwareembodiment (including firmware, resident software, micro-code, etc.) oran embodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

What is claimed is:
 1. A computer-implemented method for controllingworkload flow on converged Ethernet links, comprising: coupling, by aprocessing device, a first control loop to a second control loop, thesecond control loop monitoring the operation of the first control loop;initializing an equilibrium set point for the second control loop priorto commencing operation of the first control loop; and adjusting theequilibrium set point value in the second control loop continuouslybased on a rate of operation of the first control loop.
 2. Thecomputer-implemented method of claim 1, wherein the first control loopruns a priority flow control protocol.
 3. The computer-implementedmethod of claim 1, wherein the second control loop runs a quantizedcongestion notification protocol.
 4. The computer-implemented method ofclaim 1, wherein the initializing of the equilibrium set point valuefurther comprises establishing a low equilibrium set point value toensure that the second control loop will activate prior to the firstcontrol loop.
 5. The computer-implemented method of claim 4, wherein thelow equilibrium set point value is initialized to trigger thetransmission of a congestion notification message when a queue in thenetwork reaches a predetermined capacity.
 6. The computer-implementedmethod of claim 1, wherein the adjusting of the equilibrium set pointvalue in the second control loop further comprises dynamicallyincreasing the equilibrium set point value based on the monitoring ofthe operation of the first control loop.
 7. The computer-implementedmethod of claim 6, wherein, responsive to encountering a flag indicatinga triggering of the first control loop, pausing the increase of theequilibrium set point value.
 8. A computer program product forcontrolling workload flow on converged Ethernet links, the computerprogram product comprising: a computer readable storage medium havingprogram code embodied therewith, the program code executable by aprocessing device for: coupling, by a processing device, a first controlloop to a second control loop, the second control loop monitoring theoperation of the first control loop; initializing an equilibrium setpoint for the second control loop prior to commencing operation of thefirst control loop; and adjusting the equilibrium set point value in thesecond control loop continuously based on a rate of operation of thefirst control loop.
 9. The computer program product of claim 8, whereinthe first control loop runs a priority flow control protocol.
 10. Thecomputer program product of claim 8, wherein the second control loopruns a quantized congestion notification protocol.
 11. The computerprogram product of claim 8, wherein the initializing of the equilibriumset point value further comprises establishing a low equilibrium setpoint value to ensure that the second control loop will activate priorto the first control loop.
 12. The computer program product of claim 11,wherein the low equilibrium set point value is initialized to triggerthe transmission of a congestion notification message when a queue of aswitch in the network reaches a predetermined capacity.
 13. The computerprogram product of claim 8, wherein the adjusting of the equilibrium setpoint value in the second control loop further comprises dynamicallyincreasing the equilibrium set point value based on the monitoring ofthe operation of the first control loop.
 14. The computer programproduct of claim 13, wherein, responsive to encountering a flagindicating a triggering of the first control loop, pausing the increaseof the equilibrium set point value.